Clothing information can be used with face detection and face recognition for person clustering in images. For example, clothing clustering can be used to provide additional information to complement face-based clustering for improved person clustering in, for example but not limited to, family photos. Examples are disclosed in D. Anguelov, K. Lee, S. Gokturk, B. Sumengen, “Contextual identity recognition in personal photo albums,” CVPR 2007; J. Cui, F. Wen, R. Xiao, Y. Tian and X. Tang, “EasyAlbum: An interactive photo annotation system based on face clustering and re-ranking,” CH 2007; A. C. Gallagher and T. Chen. “Using context to recognize people in consumer images,” IPSJ Transactions on Computer Vision and Applications, 2009; J. Sivic, C. L. Zitnick and R. Szeliski. “Finding people in repeated shot of the same scene,” BMVC, 2006; Y. Song and T. Leung, “Context-aided human recognition—clustering,” ECCV 2006. A system and method for classifying an image that uses a personal clothing model developed based on clothing information in images would be beneficial.